@rfairburn when you get the chance, can you please help me come up w/ a good warning here?

So shackleton and other customers know the best practice max: https://fleetdm.slack.com/archives/C094ZAWGDHC/p1779392158155149

cc @zayhanlon

In the case you refer to there were a couple of things that at first glance happened that didn't seem intentional

• all hosts got targeted for the carve instead of a narrow target set
• the paths carved had wildcard paths in places such as /tmp or user homes that can sometime have many large files

The net effect was a load both in bandwidth and db usage that came from an X * Y * Z * A situation where:

X = number of hosts
Y = number of paths requests
Z = number of files in the paths
A = avg file size of any individual file

Any one of these can be bigger in scope usually without there being a huge issue for example:

• you can target a large number of hosts for a single small file
• you can look in several paths if you are trying to find a file/filename that only exists in 1-2 of said paths
• there can be a higher-quantiy group of small files in a single path for a single or handul of hosts
• a single large file (up to 8gb I believe) can easily be qeried

However, when all four of the dials end up being turned up (on accident in this case from the sounds of it), you suddenly end up with the db trying to write millions of new rows of metatada, while fleet is trying to populate 10s of terrabytes of data in the s3 carves bucket, while all hosts are both responding to the distributed query that does the SELECT to trigger the carve and push all of this data over the carves endpoints.

Another gotcha that did not occur in this specific case is the carves in question were only a one-off. There wasn't a scheduled query/automations=on situation, which can also compound.

The only nuance is that a particular carve select with multiple files would get combined into a single tar archive before being transferred, but we would still be subject to the 8GB size, block size of 256mb (row and s3 object per block), and the 4th-power load amplification if all factors get scaled at once.